Strong theoretical guarantees of robustness can be given for ensembles of classifiers generated by input randomization. Specifically, an $\ell_2$ bounded adversary cannot alter the ensemble prediction generated by an additive isotropic Gaussian noise, where the radius for the adversary depends on both the variance of the distribution as well as the ensemble margin at the point of interest. We build on and considerably expand this work across broad classes of distributions. In particular, we offer adversarial robustness guarantees and associated algorithms for the discrete case where the adversary is $\ell_0$ bounded. Moreover, we exemplify how the guarantees can be tightened with specific assumptions about the function class of the classifier such as a decision tree. We empirically illustrate these results with and without functional restrictions across image and molecule datasets.

Recent works have shown the effectiveness of randomized smoothing as a scalable technique for building neural network-based classifiers that are provably robust to $\ell_2$-norm adversarial perturbations. In this paper, we employ adversarial training to improve the performance of randomized smoothing. We design an adapted attack for smoothed classifiers, and we show how this attack can be used in an adversarial training setting to boost the provable robustness of smoothed classifiers. We demonstrate through extensive experimentation that our method consistently outperforms all existing provably $\ell_2$-robust classifiers by a significant margin on ImageNet and CIFAR-10, establishing the state-of-the-art for provable $\ell_2$-defenses. Moreover, we find that pre-training and semi-supervised learning boost adversarially trained smoothed classifiers even further.

A recent technique of randomized smoothing has shown that the worst-case (adversarial) l2-robustness can be transformed into the average-case Gaussian-robustness by smoothing a classifier, i.e., by considering the averaged prediction over Gaussian noise. In this paradigm, one should rethink the notion of adversarial robustness in terms of generalization ability of a classifier under noisy observations. We found that the trade-off between accuracy and certified robustness of smoothed classifiers can be greatly controlled by simply regularizing the prediction consistency over noise. This relationship allows us to design a robust training objective without approximating a non-existing smoothed classifier, e.g., via soft smoothing. Our experiments under various deep neural network architectures and datasets show that the certified l2-robustness can be dramatically improved with the proposed regularization, even achieving better or comparable results to the state-of-the-art approaches with significantly less training costs and hyperparameters.

We use the notations defined in Section 2 of the main text throughout this section. E.1 SmoothAdv Recall that a smoothed classifier ˆ f is defined from a hard classifier f: R

I thank the authors for a very well written paper with clear motivation, results and substantiation. Originality: This paper extends the work of Cohen et al. by considering smoothing distributions that can provide defenses to l0 attacks. This seems like a natural next step given the work of Cohen et al., The focus on structured classifiers (decision trees in this case) is novel. Decision trees have a natural structure which lends itself to tighter certificates for l0 attacks. The paper also provides some tricks to handle numerical issues when applying their approach to larger datasets like Imagenet.

Additional Feedback: Overall, this paper presents an efficient approach to training L2-robust models, that outperforms existing approaches in the large perturbation regime. While experiments could be improved with multiple runs, I thought they were extensive and included analyses of different design choices. Releasing code/models would help further improve reproducibility of the work. Additional comments: - Why does m have to be larger than 1? How does the method perform with m 1? - The analysis resulting in Figure 1 focuses on the log-probability gap, or logit-margin of the various classifiers. However, this is not the only factor contributing to robustness in the case of deep neural networks, which perform a highly non-linear mapping from inputs to logits; the distance to the decision boundary in input space (or input margin) is what we really care about, and is related to the logit-margin by the Lipschitzness of the mapping from input to logits; see Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks, NeurIPS 2018 for a discussion.

Overall, I believe the paper makes a meaningful empirical contribution to scalable training methods of robust classifiers. By finding adversarial examples for smoothed classifiers and modifying the training procedure, the authors significantly improve the accuracy of smoothed classifiers. Smoothed classifiers are of interest since they are scalable and come with a certificate of robustness. The paper is clearly written. However, the contribution seems incremental.

This work shows how to improve the previous state of the art for L2 robustness using smoothed classifiers (introduced by Cohen et al.) The empirical results are very strong in a very competitive area where many research groups are competing. The theoretical work, the presentation and the various technical details involved in using smoothness in PGD are all great contributions. This is an important paper in the space of adversarial ML.

Strong theoretical guarantees of robustness can be given for ensembles of classifiers generated by input randomization. Specifically, an \ell_2 bounded adversary cannot alter the ensemble prediction generated by an additive isotropic Gaussian noise, where the radius for the adversary depends on both the variance of the distribution as well as the ensemble margin at the point of interest. We build on and considerably expand this work across broad classes of distributions. In particular, we offer adversarial robustness guarantees and associated algorithms for the discrete case where the adversary is \ell_0 bounded. Moreover, we exemplify how the guarantees can be tightened with specific assumptions about the function class of the classifier such as a decision tree. We empirically illustrate these results with and without functional restrictions across image and molecule datasets.

A recent technique of randomized smoothing has shown that the worst-case (adversarial) l2-robustness can be transformed into the average-case Gaussian-robustness by "smoothing" a classifier, i.e., by considering the averaged prediction over Gaussian noise. In this paradigm, one should rethink the notion of adversarial robustness in terms of generalization ability of a classifier under noisy observations. We found that the trade-off between accuracy and certified robustness of smoothed classifiers can be greatly controlled by simply regularizing the prediction consistency over noise. This relationship allows us to design a robust training objective without approximating a non-existing smoothed classifier, e.g., via soft smoothing. Our experiments under various deep neural network architectures and datasets show that the "certified" l2-robustness can be dramatically improved with the proposed regularization, even achieving better or comparable results to the state-of-the-art approaches with significantly less training costs and hyperparameters.